Convolutional Neural Networks

The Image Problem: Why Dense Networks Struggle

Before we get to the solution, we need to understand the problem. Imagine you have a 224 by 224 pixel colour image (a common size for image models). That image has 224 × 224 × 3 = 150,528 individual pixel values. If you feed these directly into a standard dense (fully connected) layer with just 1,000 neurons, you immediately have 150 million parameters in that one layer alone.

But there is a second, more fundamental problem. A dense network has no concept of spatial structure. It treats the pixel at the top-left corner and the pixel at the bottom-right corner as completely independent, unrelated inputs. It does not know that nearby pixels tend to be related, that a curve here and an edge there combine to form an eye, or that the same object in two different positions in an image is still the same object.

Convolutional Neural Networks (CNNs) were designed specifically to solve these problems. They take inspiration from neuroscience research on the visual cortex, though it is important to note they are a mathematical engineering solution, not a precise biological model.

How Convolution Works

At the heart of a CNN is the convolution operation. The idea is beautifully simple: take a small grid of learnable numbers called a filter (also called a kernel), and slide it across the entire input image. At each position, multiply each filter value by the corresponding pixel value beneath it, sum all the results, and record a single output number. Do this at every possible position, and you get a new grid called a feature map.

In this example, the filter weights are not set by us manually. They are learned during training by backpropagation, exactly like weights in a regular neural network. The network automatically discovers what filters are useful for the task at hand.

Filters and Feature Maps

A single filter produces a single feature map. But in practice, a convolutional layer uses many filters at once. A typical first layer might use 32 filters, each one learning to detect a different low-level feature of the image.

This hierarchy from simple to complex is one of the most important properties of CNNs. Researchers at OpenAI and Google have visualised the filters learned by large image networks, and the progression from edges to shapes to objects is remarkably consistent across different models trained on different datasets.

Pooling Layers: Compressing What We Have Learned

After a convolutional layer produces feature maps, a pooling layer compresses them. The most common type is max pooling: divide the feature map into non-overlapping windows, and keep only the largest value from each window.

Pooling achieves three things at once. First, it reduces spatial dimensions (a 4×4 map becomes 2×2), which cuts computation and memory in downstream layers. Second, it provides some robustness to small translations: if a feature is detected slightly to the left or right of a window boundary, the max operation will still capture it. Third, it forces the network to summarise regional information rather than fixating on exact pixel positions.

The CNN Architecture: Putting It All Together

A complete CNN stacks convolutional layers, activation functions, and pooling layers into a pipeline that progressively extracts higher-level features, then ends with dense layers that perform the final classification.

Notice how the spatial dimensions shrink at each pooling step (28 → 13 → 5) while the number of channels (depth) increases (1 → 32 → 64). This is the classic CNN pattern: compress spatial information while extracting richer feature descriptions. The flatten layer then converts these feature maps into a regular vector that the dense layers can process.

Why CNNs Work: Two Key Principles

1. Local Connectivity

Each neuron in a convolutional layer only connects to a small local patch of the previous layer. A 3×3 filter only "sees" 9 pixels at a time. This makes sense for images: whether a pixel is part of an eye is determined by its immediate neighbours, not by pixels at the opposite corner of the image. Local connectivity dramatically reduces the number of parameters and forces the network to look for local patterns.

2. Weight Sharing

The same filter weights are used at every spatial position in the image. If the network learns a filter that detects a vertical edge, it applies that same detector everywhere, not just at one location. This means the network can recognise a vertical edge whether it appears at the top, bottom, left, or right of an image. This property is called translation equivariance: shifting the input shifts the output by the same amount.

These two properties, combined with the depth of multiple layers, are why CNNs succeeded at image recognition tasks where dense networks previously failed. They encode prior knowledge about the structure of images directly into the architecture, rather than making the network learn everything from scratch.

Landmark CNN Architectures

The history of CNNs is a story of progressively deeper and more creative architectures, each achieving a breakthrough on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) benchmark, which tests classification across 1,000 categories from 1.2 million training images.

Building a CNN in Keras: MNIST Digit Recognition

MNIST is a dataset of 70,000 handwritten digit images (60,000 training, 10,000 test), each 28×28 pixels in grayscale. It has been a standard benchmark since the 1990s. LeCun used it to evaluate LeNet-5, and it remains the "hello world" of CNNs today. A well-tuned CNN typically achieves above 99% accuracy on the test set.

import tensorflow as tf
from tensorflow.keras import layers, callbacks

# ── 1. Load and preprocess MNIST ──────────────────────────────────────
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()

# Reshape to (samples, height, width, channels) and scale to [0, 1]
X_train = X_train.reshape(-1, 28, 28, 1).astype("float32") / 255.0
X_test  = X_test.reshape(-1, 28, 28, 1).astype("float32") / 255.0

# ── 2. Build the CNN ──────────────────────────────────────────────────
model = tf.keras.Sequential([
    # Block 1: learn 32 feature maps from 3×3 patches
    # Output: (26, 26, 32) — shrinks by 2 with no padding
    layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    # Max pool halves spatial size → (13, 13, 32)
    layers.MaxPooling2D((2, 2)),

    # Block 2: learn 64 feature maps from 3×3 patches
    # Output: (11, 11, 64)
    layers.Conv2D(64, (3, 3), activation='relu'),
    # Max pool → (5, 5, 64)
    layers.MaxPooling2D((2, 2)),

    # Flatten the 5×5×64 = 1600 values into a 1D vector
    layers.Flatten(),

    # Dense classification head
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.5),              # drop 50 % of activations during training
    layers.Dense(10, activation='softmax')  # 10 digit classes, probabilities sum to 1
])

# ── 3. Compile ────────────────────────────────────────────────────────
model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',  # integer labels, not one-hot
    metrics=['accuracy']
)

model.summary()

# ── 4. Train ──────────────────────────────────────────────────────────
early_stop = callbacks.EarlyStopping(
    monitor='val_accuracy', patience=5, restore_best_weights=True
)

history = model.fit(
    X_train, y_train,
    epochs=15,
    batch_size=64,
    validation_split=0.1,
    callbacks=[early_stop]
)

# ── 5. Evaluate ───────────────────────────────────────────────────────
test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print(f"Test accuracy: {test_acc:.4f}")

Understanding the Output

A few things worth noting from the model summary. The first Conv2D layer has only 320 parameters: 32 filters × (3×3 weights + 1 bias) = 320. Despite seeing the entire 28×28 image, this layer uses far fewer parameters than a dense layer would. The second Conv2D has 18,496 parameters because it processes 32 input channels: 64 × (3×3×32 + 1) = 18,496.

The test accuracy of 99.21% means the network misclassifies only 79 of the 10,000 test images. This is a standard, reproducible result for this class of architecture on MNIST. Note that 224,034 total parameters is quite small by modern standards: large vision models used in production today have hundreds of millions to billions of parameters.

Try It in Google Colab

This code runs directly in Google Colab without any local installation. Go to colab.research.google.com, create a new notebook, enable GPU (Runtime > Change runtime type > T4 GPU), and paste the code above. Training takes under two minutes on a GPU.

Key Takeaways

• Dense networks applied to images have too many parameters and ignore spatial structure. CNNs solve both problems with convolution and pooling.
• A convolutional layer slides a small filter across the input, computing a dot product at each position. Filters are learned by backpropagation, not designed by hand.
• Each filter produces one feature map. Using many filters in parallel (32, 64, 128...) detects many different patterns simultaneously.
• Max pooling compresses spatial information by keeping only the strongest activation in each local window, reducing computation and adding robustness to small shifts.
• Weight sharing means the same filter is applied everywhere, dramatically reducing parameters. A 3×3 filter on a 1-channel input has just 9 learnable weights regardless of image size.
• CNN layers learn a feature hierarchy: early layers detect edges, middle layers detect shapes, and later layers detect high-level objects specific to the task.
• LeNet-5 (1998) established the basic CNN blueprint. AlexNet (2012) proved CNNs at scale. ResNet (2015) solved the vanishing gradient problem with skip connections, enabling networks of 100+ layers.
• A basic Keras CNN achieves 99%+ accuracy on MNIST with only 224K parameters, far fewer than a comparable dense network would need.